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ABSTRACT 

In  this  paper,  we  review  previous  work  on  both  buffered  and  unbuffered 
delta  network  performance  analysis.  We  then  analyze  the  delta  network 
performance  under  "hot  spot"  traffic  and  give  a  solution  to  the  single  "hot 
spot"  case. 

1.    Introduction 

Interconnection  networks  have  been  widely  used  in  parallel  processing  systems.  The 
performance  of  interconnection  networks  under  uniform  traffic  load  has  been  studied  with 
both  analytical  methods  and  simulations  (Patel[1981],  Dias  &  Jump[1981],  Kruskal  & 
Snir[1983],  Kumar  &  Jump[1984],  Kruskal  et  al  [1986]).  Recently,  the  "hot  spot"  traffic 
problem,  a  non-uniform  traffic  load  model  in  which  one  memory  module  is  favored,  has 
received  attention  (Pfister  &  Norton  [1985]).  In  this  paper,  we  give  an  analytical  solution 
to  network  performance  under  "hot  spot"  traffic. 


1.1.    Delta  Network 

A  typical  shared  memory  multiprocessor  system  can  be  depicted  (Fig.l)  as  N  indepen- 
dent processor  elements  (PEs)  connecting  a  set  of  memory  modules  (MMs)  through  an 
interconnection  network  (ICN).  Examples  of  such  systems  are  NYU  Ultracomputer 
(GottHeb  et  al  [1983]),  IBM  RP3  (Pfister  et  al  [1985|),  and  BBN  Butterfiy  (Rettberg  & 
Thomas  [1986]). 
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Fig.l  A  Shared  Memory  Multiprocessor  System 

Interconnection  networks  considered  here  are  packet  switching  networks  consisting  of 
multiple  stages  of  switches.  Quite  a  few  interconnection  schemes  have  been  proposed;  Feng 
[1981]  gave  a  good  survey.  Among  the  various  interconnection  networks  proposed,  those 
most  commonly  used  are  the  class  of  delta  networks  (Patel  [1981])  which  includes  the 
omega  network  (Lawrie  [1975]),  the  indirect  binary  n-cube  network  (Pease  [1975]),  and 
the  cube  network  (Siegel  &  Smith  [1978]). 

Let  an  a  xb  switch  node  be  an  a  input  and  b  output  device  which  has  the  capability  to 
route  a  message  from  any  one  of  the  a  inputs  to  any  one  of  the  b  outputs.  Let  the  outputs 
be  labeled  0,  l,...,b-l.  A  message  coming  in  an  input  terminal  is  routed  to  the  output 
labeled  d  if  the  control  digit  supplied  by  the  input  message  is  d,  where  d  is  a  base-b  digit. 
Following  Patel's  definition,  a  delta  network  is  an  a"  xb"  switching  network  with  n  stages, 
consisting  of  a  xb  switch  nodes.  The  link  pattern  between  stages  is  such  that  there  exists  a 
unique  path  of  constant  length  from  any  source  to  any  destination.  Moreover,  the  path  is  digi- 
tally controlled  such  that  a  switch  node  routes  an  input  message  to  one  of  its  b  outputs 
depending  on  a  single  base-b  digit  from  the  destination  address  of  the  message.  A  general 
delta  network  is  shown  in  Fig. 2.  An  example  of  4^  X42  delta  network  is  depicted  in  Fig. 3. 
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Fig. 2  A  General  Delta  Network. 


Fig.  3  A  4^  X42  delta  network 
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We  assume  that  the  network  terminals  on  the  PE  side  are  sources  and  the  other  ends 
(to  MMs)  are  destinations.  The  function  of  the  ICN  is  to  pass  the  requests  from  sources  to 
the  corresponding  destinations  according  to  the  request  addresses  specified  by  the  PEs. 
For  simplicity,  in  this  paper,  we  assume  the  case  a=b=k,  so  that  we  have  a  k"  xk"  square 
network.    But  usually  the  results  may  be  applied  to  general  delta  networks. 

A  network  may  be  either  unbuffered  or  buffered.  In  the  unbuffered  network,  when 
more  than  one  message  conflict  at  a  switch  node  (i.e.,  more  than  one  request  demand  the 
same  output  port  at  the  same  time)  one  of  them  will  be  chosen  to  pass  and  the  others 
rejected.  In  this  paper  we  assume  that  the  rejected  messages  are  lost.  The  sender  will 
resubmit  the  message  when  it  times  out.  The  BBN  Butterfly  uses  such  an  unbuffered  net- 
work.   Following  is  an  unbuffered  switch: 


Fig. 4  An  Unbuffered  Switch 

In  a  buffered  network,  buffer  (or  queue)  mechanisms  are  associated  with  input  ports,  output 
ports,  or  both  input  and  output  ports  of  a  switch.  Conflicts  arc  resolved  by  queueing  the 
messages.  The  Ultracomputer  and  the  RP3  use  buffered  networks.  An  output  buffering 
switch  node  which  we  will  use  for  our  analysis  is  depicted  in  the  Fig.  5. 


Fig. 5  A  Buffered  Switch 
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1.2.    Performance  Analysis  of  Interconnection  Networks 

The  work  on  analysis  of  interconnection  network  performance  could  be  classified  into 
two  models:  the  uniform  irajfic  model  in  which  we  assume  the  requests  to  the  network  are 
generated  with  their  addresses  uniformly  distributed  among  all  the  outputs;  and  the  nonuni- 
form traffic  model  in  which  the  request  addresses  are  not  uniformly  distributed.  As  men- 
tioned before,  there  are  two  types  of  switch  networks:  unbufTered  and  buffered.  They  also 
present  a  dimension  in  analyzing  network  performance. 

1.2.1.    Uniform  Traffic  Model 

In  almost  all  analysis  of  the  uniform  traffic  model,  we  not  only  assume  the  uniform 
address  distribution  but  also  assume  that  the  arrivals  of  all  requests  are  generated  as  a  ran- 
dom memoryless  process.  For  simplicity,  we  only  analyze  one  way  traffic  through  the  net- 
work. The  methods  are  usually  easy  to  apply  to  the  round  trip. 

Unbuffered  Networks 

Let  us  first  see  some  work  on  the  uniform  traffic  model.  Patel  [1981]  gave  a  recursive 
formula  to  compute  the  throughput  of  the  unbuffered  delta  network  under  given  offered 
load.  This  formula  was  also  given  in  Dias  &  Jump  [1981]  with  a  simpler  derivation. 
Assume  an  N  xN  square  delta  network  built  by  k  xk  switches  with  n=logkN  stages.  As 
shown  in  Fig. 4,  at  each  stage  i,  i=l,2,...,n,  assume  each  of  k  input  ports  of  a  switch  has 
probability  Pi-j  of  sending  a  message;  and  each  of  output  ports  has  probability  p;  of  getting 
a  message,   since  the  requests  are  uniformly  distributed  among  k  output  ports   by  the 

Pi-i 
assumptions  of  our  model.  Each  input  port  will  have  probability  to  send  a  message  to 

Pi-i  , 
an  output  port,  and  probability  ( 1  -  ;  not  to  send.  The  probability  that  all  k  input  ports 

Pi  -I   _  . 

will  not  send  messages  to  a  output  port  is  (1  -  )  .  This  is  also  the  probability  that  an 

output  port  does  not  get  message.  Therefore 

p,=l-(l-■^^     i=I,2,...,n  (1) 

Note  that  actually  po  is  the  offered  load  and  Pn  is  the  throughput.  Kruskal  &  Snir  [1983] 
showed  that  the  asymptotic  behavior  of  the  above  recursive  formula  is 
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P""0r=7P^      3(k-l)    n    ^°^n^'  ^2) 

for  large  enough  n.  Here  the  first  and  second  order  terms  are  independent  of  p,  in  which 
we  assumed  0<p^l.    In  general,  this  can  be  approximated  by 

P"'^(k-l)nV2k/p-  ^^^ 

Buffered  Networks 

The  analysis  of  buffered  networks  is  much  more  complicated  than  the  unbuffered 
case.  In  fact,  the  exact  analysis  of  buffered  networks  with  finite  buffer  space  is  still  an 
open  problem.  Assume  a  buffered  switch  is  as  shown  in  Fig. 5.  Each  queue  is  able  to 
accept  up  to  k  requests  in  one  cycle.  Assuming  infinite  buffer  size,  the  first  stage  network 
can  be  modeled  as  an  M/D/1  queue.  Based  on  this  assumption,  Kruskal  &  Snir  [1983] 
obtained  a  formula  for  the  average  queueing  delay  Wj  at  first  stage  of  the  network  with 
given  arrival  rate  p, 

There  are  several  approaches  to  calculate  the  delays  at  later  stages.  Kruskal  &  Snir 
[1983]  approximated  the  input  processes  of  later  stages  with  memoryless  processes  and 
applied  the  same  formula  (4)  to  all  stages  of  the  network.  Kruskal  et  al  [1986]  conjectured 
the  later  stage  delays  would  have  the  form 

Wod(p)*(l+ap)w,(p)  (5) 

where  p  is  the  traffic  intensity  and  a  is  some  constant.  They  then  use  simulation  to  deter- 
mine a  and  get: 

Woo*(l+^W,  (6) 

Christos  et  al  [1987]  proved  that 
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Fig. 6  Results  Given  by  Formulas  (6)  and  (7) 

We  compare  the  results  given  by  these  two  formulas  in  Fig. 6:  1.  formula  (7)  with 
k=2;  2.  formula  (7)  with  k=4;  3.  formula  (7)  with  k=8;  4.  formula  (6)  with  k=2;  5.  for- 
mula (6)  with  k  =4;  and  6.  formula  (6)  with  k  =8. 

1.2.2.    Nonuniform  Traffic  Models 

In  contrast  to  the  uniform  traffic  model,  in  nonuniform  traffic  models,  we  assume  that 
the  request  addresses  are  not  uniformly  distributed  among  all  outputs.'  Some  outputs  may 
be  preferred  over  others  by  the  requests.  One  particularly  interesting  model  is  the  "hot 
spot"  traffic  model  suggested  by  Pfister  and  Norton  [1985]. 

Kumar  and  Pfister  [1986]  analyzed  the  onset  of  the  "hot  spot"  contention.  Kruskal  et 
al  [1986]  analyze  a  single  switch  "hot  spot"  performance.  In  the  following  we  will  present  a 
theorem  for  delta  networks  under  the  "hot  spot"  traffic  model.  From  this  theorem  we  may 
reduce  the  analysis  of  the  "hot  spot"  problem  to  two  simple  cases.  Then  we  may  use  the 
same  methods  as  in  analysis  of  the  uniform  traffic  model  to  analyze  the  "hot  spot"  model. 


'The  inlerarrival  times  of  ihe  requests  could  be  uniformly  distributed.   In  fact,  as  in  the  uniform  traffic 
model,  we  usually  do  assume  a  memoryless  interarrival  lime  distribution. 
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2.    A  General  Theorem 

In  this  section,  we  present  a  model  for  the  delta  network  under  single  "hot  spot" 
trafTic  and  prove  a  general  theorem  on  such  a  model. 

Assume  an  N  xN  delta-network,  built  of  k  xk  switch  nodes.  The  network  stages  are 
numbered  l,2,...,n=logkN.  There  are  N  PEs  connected  to  the  input  side  of  the  1st  stage 
switches,  and  N  MMs  connected  to  the  output  side  of  nth  stage  switches. 

Each  PE  issues  two  types  of  requests:  uniform  requests  among  all  N  MMs,  and  a  hot 
spot  request  to  a  particular  MM.  Assume  the  uniform  requests  are  colored  white,  with 
request  rate  W;  and  the  hot  spot  requests  are  colored  red,  with  rate  R.  Each  PE  issues 
p=W  +  R  requests  per  cycle.  Following  the  paths  of  the  hot  spot  requests,  we  have  a  red 
traffic  tree  which  is  rooted  at  the  hot  MM,  and  spans  N  PEs  as  leaves. 

We  label  each  input  and  output  of  a  switch  with  its  traffic  rate.  The  labeling  scheme  is 
as  follows: 

At  stage  1:  Initially,  we  label  all  inputs  po,o;  the  outputs  in  the  red  traffic  tree  are 
labeled  p,  j ;  those  not  in  the  red  traffic  tree  poi. 

At  stage  i,l  <i  ^n.  We  have  have  following  two  labeling  rules: 

(1)  If  all  inputs  of  a  switch  are  labeled  p,-i,i-i,  then  the  output  which  leads  to  the  hot 
MM  is  labeled  p,_i;  the  other  k-1  outputs  Pi-i,i.    Such  a  switch  will  be  in  the  hot  traffic  tree. 

(2)  If  all  inputs  of  a  switch  are  labeled  Pj.i-i,j  <i- 1,  then  all  k  outputs  are  labeled  pjj. 

Suppose  we  start  the  labeling  process  according  to  the  above  labeling  scheme  and  stop 
only  when  there  are  no  labeling  rules  applicable. 

When  the  labeling  process  stops  there  are  two  possible  outcomes:  all  network  links  are 
labeled;  or  some  network  links  are  not  labeled.  If  the  labeling  process  stops  without  all  net- 
work links  labeled  there  must  be  a  switch  node  which  has  different  input  labels.  Otherwise 
we  could  apply  either  rule  (I)  or  (2)  to  proceed  the  labeling  process.  On  the  other  hand,  if 
all  network  links  are  labeled  at  the  end  of  the  labeling  process,  all  input  ports  of  the  same 
switch  node  must  have  the  same  label.  In  fact,  all  network  links  can  be  labeled  with  the 
above  labeling  rules,  as  shown  below. 
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(a)  Switch  nodes  in  the  hot  trafTic  tree;  and 

(b)  Switch  nodes  not  in  the  hot  traffic  traffic  tree. 


Fig.  7  Two  basic  cases  in  the  hot  spot  traffic  model: 

Theorem:  Under  the  above  definition,  each  switch  node  in  a  delta  network  has  all  its 
inputs  labeled  with  the  same  Pj.j-i  where  i  is  the  stage  of  the  node,  and  j  <i. 

Note  that  i  - 1  =j  if  and  only  if  the  node  is  in  the  red  traffic  tree. 

Proof  of  the  theorem: 

Without  loss  of  generality,  we  assume  that,  a  message  is  rooted  to  output  a, 
O^a^k-1,  at  stage  i  if  the  i'*^  digit  of  the  k-based  destination  address  is  a.  Then  we  have 
the  following  Lemma. 

Lemma:  All  traffic  routed  to  the  same  switch  node  at  stage  i  will  have  their  first  i  — 1 
digits  of  the  k-based  destination  address  the  same. 

The  lemma  follows  from  our  assumptions  that  there  is  a  unique  constant  length  path 
from  any  source  to  any  destination  and  the  path  is  specified  by  a  k-based  destination 
address.  For  any  two  requests  meeting  at  stage  i,  if  their  last  n-i-i-1  digits  of  destination 
address  were  the  same  they  would  end  up  at  the  same  destination.  So  they  must  have  their 
first  i-1  digits  of  destination  address  the  same.  Otherwise  wc  would  not  have  a  unique 
destination  address. 

Then  we  prove  the  theorem  by  induction: 

At  stage  1,  it  is  true  from  the  labeling  definition. 

Assume  it  it  true  at  stage  i,  but  not  at  stage  i  +  1,  which  means  that  there  is  a  switch  at 
stage  i-f-l  whose  2  inputs  labeled  Pjj  and  pu  such  that  j ''^I. 

Then  from  the  labeling  definition,  we  know  that  j  ^i  and  1  ^i.  Without  loss  of  general- 
ity, we  assume  j>l.  The  input  labeled  pj  j  means  that  this  request  has  a  destination  address 
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with  j  digits  the  same  as  the  hot  spot  address.  The  pij  request  has  a  destination  address 
with  1  digits  the  same  as  the  hot  spot  address,  but  not  the  (l  +  l)st  digit.  Since  j^  +  l,  pj  j 
and  pj  i  traffic  flows  differ  at  their  (I  +  l)st  digit.  On  the  other  hand,  pjj  and  pij  meet  at 
stage  i  +  1.  From  the  Lemma,  they  should  have  their  first  i  digits  same.  This  contradicts 
that  they  differ  at  the  (l  +  l)st  digit.  QED. 

The  property  stated  by  the  theorem  is  really  nice.  It  shows  that  under  the  single  "hot 
spot"  traffic  model  we  assumed,  the  analysis  of  network  performance  could  be  reduced  to 
two  simple  cases  (corresponding  to  the  two  labeling  rules):  (1)  those  switches  which  are 
located  in  the  hot  traffic  tree;  and  (2)  those  switches  which  are  not  in  the  hot  traffic  tree. 
For  both  cases  all  inputs  of  the  same  switch  node  have  the  same  history.  Therefore  they 
have  the  same  distribution.  The  switches  in  case  (2)  have  the  same  properties  as  the 
switches  in  the  uniform  traffic  model  and  can  be  analyzed  with  the  same  methods.  For 
case  (1),  we  have  one  output  port  preferred  by  all  the  input  ports  in  a  switch,  which  needs 
a  special  treatment,  but  no  fundamental  difficulty.  Fig. 8  is  a  labeled  2-'  x2^  delta  network. 

Po.O   I 1 Po.l  I 1  Po,2    I 1  Po,3 

Po.o 


Fig. 8  A  labeled  2^  x2^  delta  network 
with  hot  location  at  MM  #  3 

Furthermore,  from  the  theorem,  we  may  draw  a  picture  for  the  network  under  the  sin- 
gle "hot  spot"  traffic  model.  At  the  1st  stage,  all  the  inputs  are  colored  red  and  labeled 
Po,o-  But  each  switch  will  only  have  one  of  its  outputs  colored  red;  the  total  number  of  N/k 
output  ports  will  be  labeled  red  with  pij.  The  remaining  N(k-l)/k  output  ports  will  be 
labeled  white  with  po,i.  Remember  that  the  theorem  tells  us  that  only  like  labeled  outputs 
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of  the  1st  stage  will  go  to  the  inputs  of  the  same  switch  at  the  2nd  stage.  Therefore,  at  the 
2nd  stage,  we  have  N/k^  switches  with  their  inputs  labeled  pij  and  colored  red.  The 
remaining  N(k-l)/k^  switches  all  have  inputs  labeled  po.i-  Then  we  apply  the  labeling 
rules  to  the  2nd  stage,  and  continue,  stage  after  stage.  The  result  is  stated  in  following 
corollary. 

Corollary:  Following  our  labeling  rules  above,  at  stage  j,  1  ^j  ^n,  the  inputs  of  N/RJ 
switches  are  labeled  Pj-ij-i;  and  the  inputs  of  N(k- l)/k''"^')  switches  are  labeled  Pi,j-i, 
for  0:^i<j-l.  Then  there  are  N/k'  outputs  labeled  pj.j;  and  N(k-l)/k('^')  outputs  labeled 
Pi.j,  for  0^i<j. 

We  are  particularly  interested  in  the  last  stage.  At  stage  n  (last  stage),  we  have  one 
output  labeled  pn.n,  and  N(k- l)/k'*  +  ')  outputs  labeled  p,.n  for0^i<n. 

3.    Unbuffered  Network 

Now  we  can  apply  the  theorem  proved  above  to  solve  the  problem  of  unbuffered  net- 
work performance  under  a  single  "hot  spot"  traffic  model.  Since  there  are  only  two  basic 
cases,  we  examine  each  in  detail.  Assume  that  a  label  of  a  switch  represents  the  probabil- 
ity that  a  message  will  be  sent  through  the  link  that  label  is  tagged.  At  any  stage  j,  1  ^j  ^n, 
all  switches  fall  in  one  of  two  cases. 

In  the  first  case  (See  Fig. 7a),  all  k  inputs  of  a  switch  are  labeled  pj_,_j_,.  One  of  the  k 
outputs  is  the  "hot"  output  labeled  Pj  j  and  the  others  are  labeled  pj-,j  which  are  white 
traffic  only.  This  is  the  switch  node  the  red  traffic  (i.e.,  "hot"  traffic)  goes  through. 
Assume  that  the  red  traffic  ratio  is  rj-,.  Therefore,  within  the  Pj-ij-i  traffic  presented  at 
the  switch  input,  there  is  rj-,Pj-,j-,  red  traffic  and  ( 1 -rj-i)pj-i,j-i  white  traffic.  Note 
that  the  red  traffic  is  only  destinated  to  the  "hot"  output  labeled  pjj  and  the  white  traffic  is 
uniformly  distributed  to  all  k  outputs  of  the  switch.  We  first  show  how  to  compute  the 
"hot"  output  pj,j.  The  "hot"  output  consists  of  the  red  traffic  part  and  the  white  traffic 
part.  Each  of  k  inputs  has  probability  rj_iPj-],j-i  of  sending  red  traffic  to  the  "hot"  out- 
put. On  the  other  hand  the  uniform  white  traffic  of  an  input  port  has  probability 
(l-rj-i)pj-,,j-,/k  of  sending  a  message  to  each  of  k  outputs.  The  probability  that  none  of 
k  inputs  send  any  traffic  to  the  "hot"  output  is  [  1 -rj-iPj-)j-i -( 1 -rj-i)pj-i,j-i /k]'*. 
Therefore  we  have  the  formula  of  the  probability  that  the  "hot"  output  has  a  message: 
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Pj.j=l-l(l-rj-,Pj-,.i-,-(l-rj-,)pj-,.j-,/k]'<.  (8) 

Following  a  similar  argument  we  know  that  the  probability  a  white  output  has  a  message  is: 

Pj-i.j=l-ll-(I-rj-.)Pj-,.j-,/k]'^  (9) 

where, 

rj=— '^^'^ j=0,l n.  (10) 

Remember  that  R+W=po,o,  where  R  is  the  request  rate  to  the  "hot  spot"  and  W  is  the 
uniform  traffic  rate.  With  fair  random  resolution  of  collisions  at  outputs,  equation  (10)  is 
obtained  with  simple  calculations. 

In  the  second  case  (see  Fig. 7b),  when  the  switch  node  is  not  in  the  red  traffic  tree,  all 
the  inputs  are  labeled  p,j_,  at  stage  j,  for  1  ^j  ^n,  0^i<i-l.  By  the  assumption  on  the 
white  traffic,  all  the  input  requests  are  uniformly  distributed  among  k  outputs  which 
labeled  Pij.  Therefore  we  have 

P..j=l-(l-P..j-i/k)''-  (11) 

Take  a  4^  ^4'  (jgj^a  network  for  example.  In  the  "hot  spot"  traffic  model,  at  the  out- 
puts of  last  stage,  traffic  could  be  divided  into  6  classes: 

p(5,5):  "hot"  traffic,  one  output  only; 

p(4,5):  the  traffic  confiicts  with  the  "hot"  traffic  for  4  stages,  3  outputs; 
p(3,5):  the  traffic  confiicts  with  the  "hot"  traffic  for  3  stages,  12  outputs; 
p(2,5):  the  traffic  conflicts  with  the  "hot"  traffic  for  2  stages,  48  outputs; 
p(l,5):  the  traffic  conflicts  with  the  "hot"  traffic  for  1  stage,  192  outputs;  and 
p(0,5):  the  trafllc  has  no  conflict  with  the  "hot"  traffic,  768  outputs. 

The  effects  of  "hot"  traffic  on  the  other  five  classes  of  traffic  are  shown  at  Fig.  8  (with 
R/Po.o  =  10%)  and  Fig. 9  (with  R/po.o  =  5%). 
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Fig.9  A  4^  x4'  delta  network  performance 
under  10%  hot  spot  offered  load 
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Fig.  10  A  4'  x4^  delta  network  performance 
under  5%  hot  spot  offered  load 
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4.    Buffered  Network 

As  we  mentioned  before,  the  problem  of  finite  buffer  space  switch  analysis  is  a  hard 
problem,  not  yet  solved.  With  the  assumption  of  infinite  buffer  space  we  have  some  solu- 
tions to  the  problem  (Kruskal  &  Snir  [1983],  Kruskal  et  al  (1986)). 

Under  the  "hot  spot"  traffic  model,  if  we  assume  infinite  buffer  space,  then  the  net- 
work is  not  blocking.  Assume  that  the  request  rate  is  p=W  +  R,  where  W  is  the  uniform 
request  rate  and  R  is  the  "hot  spot"  request  rate.  Since  the  network  is  not  blocking,  every 
request  is  able  to  get  into  the  network,  if  the  network  has  a  stable  state  for  a  given  offered 
load.  When  it  reaches  the  stable  state,  at  every  switch  node,  the  total  input  rate  should  be 
equal  to  the  output  rate.  We  then  have 

p,.j=W  +  Rk',i=0,l,...,n  (12) 

and 

Pj.i=W,  i>j,i=l,2 n.  (13) 

In  order  for  the  network  to  be  stable,  we  should  have 

W+NR^l.  (14) 

Equations  (12)  and  (13)  are  two  basic  conditions  for  the  network  in  stable  state. 
Applying  the  techniques  in  Kruskal  et  al  [1986]  we  may  obtain  the  delay  information  for 
the  requests  in  the  network.  Due  to  the  restriction  of  equation  (14)  we  may  only  handle  a 
very  small  amount  of  "hot  spot"  requests  without  combining  (Gottlieb  et  al  [1983]). 
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